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We propose a new model of neural network. It consists of spin variables to describe 
the state of neurons as in the Hopfield model and new gauge variables to describe 
the state of synapses. The model possesses local gauge symmetry and resembles 
lattice gauge theory of high-energy physics. Time dependence of synapses describes 
the process of learning. The mean field theory predicts a new phase corresponding 
to confinement phase, in which brain loses ablility of learning and memory. 

1 Introduction 

The Hopfield model of neural network succeeds to explain some basic 
functions of human brain such as associated memory. However, to be a more 
realistic model, at least the following points should be taken into account; 

(1) Effects of external stimulations through eyes and ears on neurons. 

(2) Effects of time variations of synapses on neurons. 

The point (2) is essential to describe the function of learning, since the possible 
patterns to memorize are completely determined according to the strengths 
of synapse connections among neurons as long as they are time independent. 
Their time dependence induces the process of learning itself. 

In Sect. 2, we review the Hopfield model briefly. In Sect. 3, we propose a 
new model of neural network, in which the strengths of synapse connections 
are regarded as gauge connections and vary in time according to the gauge 
principle. By using the mean field theory, we see that the model predicts a 
new state of brain in which both learning and memory are impossible. In 
Sect. 4, we present future outlook. 

2 Hopfield model 

Let us review the framework of the Hopfield model briefly. Its energy 
En({Si}) is given by 



N N 

E H ({S i }) = --J2^2JijSiS j , (1) 



i=l j=l 
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where Si = ±1 is the Ising spin variable to describe the state of i-th neural 
cell (i = 1,2, ■■■,N) as Si = 1; excited, Si = —1; unexcited. Jij is a given 
real constant that expresses the strength of synapse connection for the signal 
propagating from the j-th cell to the i-th cell. 

The time evolution of Si(t) for every discrete time interval e (often set 
unity) is governed by the following equation; 

Si(t + e) = sgn [-^f(t)] = sgn{Y, JijSj(t)}. ( 2 ) 

1 3 

Thus, Jij > tries to proliferate (un)excited cells, while Jij < prefers 
mixtures of excited and unexcited ones. If the system converges into certain 
configuration of {Si} after a sufficiently long time, it corresponds to recalling 
certain pattern. Such a configuration should be a stationary point of Eh, i.e., 
dEn/dSi = for every i. All these configurations are determined once Jij 
are given. 

Practically speaking, the rule (jfy may not necesssarily hold all the time 
interval due to anavoidable error in signal propagations. Such a situation 
may be simulated by adding random noises Tji(t) into the square bracket in 
the right-hand side of (^|), whose strength can be identified as a fictitious 
" tempeteture" T. If T is large, the error in signal propagations occurs fre- 
quently. Thus it is interesting to study statistical mechanics of the system 
Eh by using Boltzmann distribution. The partition function Zh is given by 

Zh = H ]T eM-PE H ), P = l/T. (3) 

i S^=±l 

In case that all Jij are positive, the system has two phases; 

- Ferrofagnetic phase below certain critical temperature T c , T < T c , in which 
there is a long-range order and the average (Si) ^ 0. 

- Paramagnetic phase above T c , T > T Cl in which Si are random and (Si) = 0. 

The ferromagnetic phase corresponds to the state of clear memory, while in 
the paramagnetic phase no definite patterns can persist. If J^ is complicated, 
there arises a spin-glass phase as we shall see. 

Explicitly, let us fix Jy according to the Hebb's rule as 

1 M 

Q=l 
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T Paramagnetic 




Figure 1. Phase structure of the Hopfield model in a{= M/N) — T plane. 



where we prepare M patterns Si = £f (a = 1, • • • M) to-.recall. The replica 
method gives rise to the phase diagram shown in Fig.lE Each phase is ex- 
plained in Table 1. 



Table 1. Phases of the Hopfield model. 



Phase 






Property 


Ferromagnetic 






memory 


Spin glass 







false memory 


Paramagnetic 








no memory 



3 New model with local gauge symmetry 

As pointed out in Sect.l, to incorporate the function of learning, one needs 
the time variation of Jjy. There are various approaches for this point. Below, 
we regard both Si and Jjj as dynamical variables and treat them on an equal 
footing. Let us assume that their time dependence is controlled so as to 
reach a local minimum of the new energy E({Si, Jy})- To determine E, we 
impose the conidtion that E is local gauge invariant under the following gauge 
transformation; 

Si-tS'^ViSi, J,, J',, \,J,,\[ r E({S' i ,Jl j })=E{{S i ,J ij }),(5) 

where Vi — ±1 is the Z(2) variable associated with z-th cell. Since Jy describes 
the state of the synapse connecting i-th and j-th cells, it is natural to regard it 
as the connection of gauge theory. The neural network may possesses certain 
conservative quantity in association with the long-term memory. The local 
gauge symmetry we address may respect such a conservation law. This point 
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Figure 2. Graphical representation of each term in E of (g). 

will be reported in detail in a separate publication!] It is often stressed that 
the connections Jjy and Jjj are independent (asymmetric). Then a general 
form of E({Si, Jij}) and the partition function Z may be given by 

E = — — Si Jij Sj + — Jij Jij 

+ ^| ^ JijJjkJki + ^jj ^ JijJjkJkiJli +•'• 
i,j,k,£ 

z =u e n / ^ ex p(-^)- = i/r. (6) 

Each term of _E is gauge invariant since = 1, and depicted in FigJ|. 

_E takes a form very similar to the lattice gauge theory in particle physics, 
where Sj corresponds to a matter field and Jij to an exponentiated gauge field. 
If the parameters <?2, <?3, <?3, ■■• are set zero, E reduces to Eh of (Q). 

3.1 Model I 

To be explicit, we need to specify the model further. Let us first consider the 
Z(2) Higgs gauge model on a 3D cubic lattice, 

3 1 

X /.I—I X fl<V 

Z i = n E II E eM-PEi) = eM~m), (7) 

where x denotes the lattice site on which S x lives, and \i (= 1,2,3) denotes 
both the direction and the unit vector. We consider only the connections 
between the nearest-neighbor sites (x, x + li) and treat them as symmetric 
Z{2) variable on a link (x,x + (J,); J x .x+n = Jx+n,x = Jxn = ±1- The A term 
and the l/g 2 -term is depicted in Fig.[| 
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X + V 



x x + n 
(b) 

Figure 3. Graphical representation of E of (a) A term, (b) g~ 2 term. 

The time evolution of J xfl may be given by the similar rule as @), 

3 Ei 

S x {t + e) = sgn[- — (t) + r] x (t)}, 

J x „(t + ae) = sgn[—^-{t) + (8) 

where a sets the ratio x>f the two time scales for S x and J xfi . We report our 
study of (||) elsewhere. □ 

Below we study the phase diagram of E\ by using the mean field theory, 
which is formulated as a varational principle as follows. Let us introduce a 
variational energy Eq . Then the Jensen-Peierls inequality gives rise to 

Fi <F + (£i-£o)o, 
Z = Tr exp(-(3E a ) = exp(-/3F ), 
(O) = Z^ 1 Tr O expi-pEo), (9) 

where Tr implies Y\ x Yls x =±i Tlx ^Ej, =±r We crioose the variational pa- 
rameters in Eo so that the right-hand-skle of inequality reaches the minimum. 
For Eq we assume the translational invariance of mean fields and employ the 
single-site and single-link energy, 

X fl X 

with the two variational parameters, W xfi = W and h x = h. The result is 
given in Table 2 and Fig.|| 



Table 2. Phases of Model I of @). 



Phase 


(Sx) 


(J*?) 


Memory 


Learning 


Hopficld Model 


Higgs 






yes 


yes 


Ferromagnetic 


Coulomb 







no 


yes 


Paramagnetic 


Confinement 








no 


no 


not available 



x + n 



(a) 
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1 st Order 




Figure 4. Phase diagram of Model I of ([ij). The point marked as Ising locates the second- 
order transition point of the Ising model. The point PG locates the first-order transition 
point of the Z(2) pure gauge theory. 



As shown in the Table 2, one may take (S x ) as an order parameter to judge 
whether the system succeeds to recall definite patterns, and (J X /j) to judge 
whether the system is able to learn some new patterns. In the confinement 
phase, neither memory nor learning is possible. This phase is missing in the 
Hopfield model. 

We note that Monte Carlo simulations of the 3D Z(2) Higgs gauge model 
exhibit these three phases, but the phase boundary of Higgs and confinement 
phases does not continue to (3/g 2 = but terminates at some finite value. 
These two phase can be reached each other smoothly by contouring the end 
point. This "complementarity" reflects that \J xt i\ — 1 and is proved by rigor- 
ous treament,u but not predicted correctly in our variational treatments 

To what extent are these results trustworthy? To answer this question, 
we intoroduce and study other two models, Model II and Model III. 



3.2 Model II 

The framework and the energy E\\ of Model II is same as Model I, but we 
allows additional state J xfJ , = 0, which describes the possibility that the con- 
nection between x and x + fi is missing; 

E u = E h Z n = n E II E exp(-/3£„). (11) 

x S x =±lx,fJi J xfl =0,±l 

The phase diagram calculated by the similar mean field theory is shown 
in Fig|| The global structure remains the same as Fig.^, although the region 
of the confinement phase is enlarged as expected since the added states clearly 
favor this phase. 
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Figure 5. Phase diagrams of Model II of (|ll| 



3.3 Model III 

In Model III, we intorduce two independent Z(2) variables J xfl and J xfl for 
the synapse between x and x + fi to take their independence into account as 

We also define J x .-^ = J x -^,x- The energy Em is then given by 



— — J ^ ' ^ ' \_JxfiJx+n,vJx+v,fj,Jxi' + ")] 



(13) 



We note that the expression JijSiSj in £?h, i, ii washes out the asymmetry 
Jij ^ Jji, while the fist term of (|l^) reflects it. Each term in E is depicted in 
Fig.6. 



'X)_l 



X + V 



_• X + jl 



JXfl 



(a) 



x -\- V 



x + n x x + A 1 
(b) (c) 



Figure 6. Graphical representation of Model III of Jl3|), (a) and J x ^\ (b) A term; (c) 
g — 2 term. 
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Figure 7. Phase diagrams of Model III of (|l3|). 



The phase diagram in mean field theory is shown in Fig.^. The global 
structure still remains unchanged, although the region of the confinement 
phase is diminished considerably. This may be understood since the first term 
in Em is bilinear in in contrast with En, i. n, and favors nonvanishing Jy. 

4 Summary and outlook 

Our results may be summerized with some future outlook as follows; 

- Due to the dynamical variables Jij , a new (confinement) phase appears at 
high temperatures, which describes the new state of no ability of learning 
and memory. 

- To decribe the spin-glass phase, further study of long-range correlation 
and/or frustrations is necessary. 

- Relaxing of J^ = ±1(, 0) to — oo < Jy < oo may be interesting, but requires 
a detailed form of the energy. 

- Study of the time evolution of Jij and Si may describe the mechanism of 
learning such as the process to forget the patterns. 

- Study of the effect of local gauge symmetry on brain function on a " quan- 
tum" leveLis interesting. Introduction of gauged versions of quantum brain 
dynamics and cellular automata with Penrose's ideaS may be the first step. 
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